Detecting and responding to sirens

ABSTRACT

The technology relates to detecting and responding to emergency vehicles. This may include using a plurality of microphones to detect a siren noise corresponding to an emergency vehicle and to estimate a bearing of the emergency vehicle. This estimated bearing is compared to map information to identify a portion of roadway on which the emergency vehicle is traveling. In addition, information identifying a set of objects in the vehicle&#39;s environment as well as characteristics of those objects is received from a perception system is used to determine whether one of the set of objects corresponds to the emergency vehicle. How to respond to the emergency vehicle is determined based on the estimated bearing and identified road segments and the determination of whether one of the set of objects corresponds to the emergency vehicle. This determined response is then used to control the vehicle in an autonomous driving mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/476,830, filed Sep. 16, 2021, which is a continuation ofU.S. patent application Ser. No. 16/843,928, filed Apr. 9, 2020, nowissued as U.S. Pat. No. 11,164,454, which is a continuation of U.S.patent application Ser. No. 16/392,745, filed Apr. 24, 2019, now issuedas U.S. Pat. No. 10,650,677, which is a continuation of U.S. patentapplication Ser. No. 15/689,336, filed Aug. 29, 2017, now issued as U.S.Pat. No. 10,319,228, which claims the benefit of the filing date of U.S.Provisional Patent Application No. 62/525,423, filed Jun. 27, 2017, theentire disclosures of which are incorporated by reference herein.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousmode where passengers may provide some initial input, such as a pickupor destination location, and the vehicle maneuvers itself to thatlocation.

In order to do so safely, these vehicles must be able to detect andidentify objects in the environment as well as respond to them quickly.This is especially true in situations involving emergency vehicles. Todetect such vehicles using visual cues can be extremely difficult,especially as these vehicles can differ dramatically, and in manysituations may actually resemble non-emergency vehicles. Moreover, ifflashing lights are not readily discernable by the autonomous vehicle'sperception system, such as when an emergency vehicle is occluded or outof range of the vehicle's perception system, detection can be nearlyimpossible. One common feature of emergency vehicles are the use ofsirens to signal an emergency or a need for the emergency vehicles topass through traffic quickly. However, in some cases, an emergencyvehicle may not be directly observed or identified, even where itspresence is identified through its siren. In such cases, it is criticalfor the autonomous vehicle's reaction that the inferring the emergencyvehicle's position, bearing, velocity, and intent through detection andtracking of the siren.

BRIEF SUMMARY

One aspect of the disclosure provides a method of detecting andresponding to emergency vehicles. The method includes using, by one ormore processors, a plurality of microphones arranged at differentlocations on a vehicle to detect a siren noise corresponding to anemergency vehicle; using, by the one or more processors, output from theplurality of microphones to estimate a bearing of the emergency vehicle;comparing, by the one or more processors, the estimated bearing to mapinformation identifying locations of roadways subdivided into roadsegments in order to identify one or more road segments which theemergency vehicle is traveling; determining, by the one or moreprocessors, how to respond to the emergency vehicle based on theestimated bearing and the identified one or more road segments; andcontrolling, by the one or more processors, the vehicle in an autonomousdriving mode based on the determination of how to respond to theemergency vehicle.

In one example, the method also includes using output from the pluralityof microphones, estimating a range of the emergency vehicle and whereindetermining how to respond to the emergency vehicle is further based onthe estimated range. In this example, the method also includes usingoutput from the plurality of microphones over time, estimating avelocity of the emergency vehicle and wherein determining how to respondto the emergency vehicle is further based on the estimated relativevelocity. In another example, the method also includes using output fromthe plurality of microphones over time, estimating a velocity of theemergency vehicle and wherein determining how to respond to theemergency vehicle is further based on the estimated relative velocity.In another example, controlling the vehicle includes changing from afirst lane to a second lane. In another example, controlling the vehicleincludes pulling the vehicle over onto a shoulder area. In anotherexample, controlling the vehicle includes continuing on a currenttrajectory of the vehicle. In another example, controlling the vehiclefurther includes decreasing a speed of the vehicle.

Another aspect of the disclosure provides a method of detecting andresponding to emergency vehicles. The method includes using, by one ormore processors, a plurality of microphones arranged at differentlocations on a vehicle to detect a siren noise corresponding to anemergency vehicle; using, by the one or more processors, output from theplurality of microphones to estimate a bearing of the emergency vehicle;receiving, from a perception system of the vehicle, informationidentifying a set of objects in the vehicle's environment as well ascharacteristics of the set of objects; determining, by the one or moreprocessors, whether one of the set of objects corresponds to theemergency vehicle based on the characteristics of the set of objects;determining, by the one or more processors, how to respond to theemergency vehicle based on the estimated bearing and the determinationof whether the one of the set of objects corresponds to the emergencyvehicle; and controlling, by the one or more processors, the vehicle inan autonomous driving mode based on the determination of how to respondto the emergency vehicle.

In one example, the characteristics include an estimated objectposition, and determining whether one of the set of objects correspondsto the emergency vehicle is further based on a comparison between thecharacteristics of the set of objects and the estimated bearing. Inanother example, the method also includes using output from theplurality of microphones over time, estimating a range of the emergencyvehicle, and wherein the characteristics include an estimated objectposition, and determining whether one of the set of objects correspondsto the emergency vehicle is further based on a comparison between thecharacteristics of the set of objects and the estimated range. Inanother example, the method also includes using output from theplurality of microphones over time, estimating a velocity of theemergency vehicle, and wherein the characteristics include an estimatedobject velocity, and determining whether one of the set of objectscorresponds to the emergency vehicle is further based on a comparisonbetween the characteristics of the set of objects and the estimatedrelative velocity. In another example, the method also includescomparing, by the one or more processors, the estimated bearing to mapinformation identifying locations of roadways subdivided into roadsegments in order to identify one or more road segments which theemergency vehicle is traveling, and wherein determining how to respondto the emergency vehicle is further based on the estimated bearing andthe identified one or more road segments. In another example, the methodalso includes identifying a first likelihood that each given object ofthe set of objects is the emergency vehicle based on the characteristicsof that given object, and wherein determining whether one of the set ofobjects corresponds to the emergency vehicle is further based on anyfirst likelihoods. In another example, the method also includesidentifying a second likelihood that each given object of the set ofobjects is not the emergency vehicle based on the characteristics ofthat given object, and wherein determining whether one of the set ofobjects corresponds to the emergency vehicle is further based on anysecond likelihoods. In another example, the method also includescontrolling the vehicle includes stopping at an intersection when thevehicle would otherwise have right of way to proceed through theintersection.

A further aspect of the disclosure provides a system for detecting andresponding to emergency vehicles. The system includes one or moreprocessors configured to use a plurality of microphones arranged atdifferent locations on a vehicle to detect a siren noise correspondingto an emergency vehicle; use output from the plurality of microphonesover time, estimating a bearing of the emergency vehicle; compare theestimated bearing to map information identifying locations of roadwayssubdivided into road segments in order to identify one or more roadsegments which the emergency vehicle is traveling; receive, from aperception system of the vehicle, information identifying a set ofobjects in the vehicle's environment as well as characteristics of theset of objects; determine whether one of the set of objects correspondsto the emergency vehicle based on the characteristics of the set ofobjects; determine how to respond to the emergency vehicle based on theestimated bearing and the identified one or more road segments and thedetermination of whether one of the set of objects corresponds to theemergency vehicle; and control the vehicle in an autonomous driving modebased on the determination of how to respond to the emergency vehicle.

In one example, the one or more processors are also configured to useoutput from the plurality of microphones to estimate a range of theemergency vehicle and wherein determining how to respond to theemergency vehicle is further based on the estimated range. In anotherexample, the one or more processors are further configured to use outputfrom the plurality of microphones to estimating a velocity of theemergency vehicle and wherein determining how to respond to theemergency vehicle is further based on the estimated relative velocity.In another example, the system also includes the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withaspects of the disclosure.

FIG. 2 is an example representation of map information in accordancewith aspects of the disclosure.

FIGS. 3A-3D are example external views of a vehicle in accordance withaspects of the disclosure.

FIG. 4 is an example view of a section of roadway corresponding to themap information of FIG. 2 in accordance with aspects of the disclosure.

FIG. 5 is an example representation of bearing data in accordance withaspects of the disclosure.

FIG. 6 is an example representation of range data in accordance withaspects of the disclosure.

FIG. 7 is an example representation of relative velocity data inaccordance with aspects of the disclosure.

FIG. 8 is a flow diagram in accordance with aspects of the disclosure.

DETAILED DESCRIPTION Overview

The technology relates to autonomous vehicles for transporting peopleand/or cargo between locations. In order to address the situationsdescribed above, in addition to the perception system which uses lasers,radar, sonar, cameras or other sensors to detect objects in theenvironment, the autonomous vehicle may be equipped with a series ofmicrophones or microphone arrays arranged at different locations on thevehicle. These microphones may be used, as discussed below to detect andidentify emergency vehicles providing a way to gain awareness and reactto an emergency vehicle when it is occluded or not visible to thevehicle's perception system as well as an independent and/or redundantway to detect an emergency vehicle when the emergency vehicle is visibleor detectable by the vehicle's perception system.

The output of these microphones may be input into a model in order todetect potential emergency vehicle sirens. Once a siren noise isdetected by the model, the timing of the siren noise reaching each ofthe microphones may be used as input to a second model in order toprovide measurements as to a likely bearing, or relative direction, ofthe source of the siren, or rather a probability distribution overpossible bearings. In addition, the siren noise, amplitude, and timingmay be input into a third model to provide a probability distributionover possible ranges of the source of the siren. A fourth model may beused to estimate a probability distribution over possible velocities ofthe source of the siren noise.

The information from the models may be provided to one or more computingdevices of the vehicle. These computing devices may use the estimatedbearing, estimated range, and estimated relative velocity to determinehow the vehicle should react to the vehicle. However, to increase theusefulness of the response, the information provided by the models maybe compared to objects detected in the vehicle's environment todetermine whether any of those objects are the source of the sirennoise. Once a particular vehicle is identified as the source of thesiren noise, the vehicle may be identified as an emergency vehicle. Atthis point, the observed movements of this emergency vehicle may also beconsidered when determining how best to respond to the emergencyvehicle, thereby further improving the usefulness of the response.

In addition to comparing the model output to information from theperception system, the estimated bearing, estimated range, and estimatedrelative velocity may be compared to map information describing roadwayfeatures in the vehicle's environment. This may be used to identify alikely roadway, road segment or, in some cases, even a specific lane inwhich the emergency vehicle is traveling, again, even where the sourceis out of the range of the vehicle's perception system or otherwiseoccluded. The location of the vehicle relative to the emergency vehicle(and vice versa) may be a significant factor in determining what type ofresponse is appropriate.

The features described herein may allow an autonomous vehicle to detect,identify, and respond to emergency vehicles even when those emergencyvehicles are not readily detectable by the vehicle's perception system.When a siren sound is detected, the use of multiple locations for themicrophones may allow the vehicle's computers to not only detect asiren, but also to estimate a relative direction, heading and velocityof the source of the siren. This may provide the computing devices ofthe vehicle with critical information for determining how to react tothe siren noise. In addition, the comparison of the direction, headingand velocity of the source of the siren to identified vehicles and mapinformation may allow the computing devices to further improve theresponse of the vehicle.

Example Systems

As shown in FIG. 1 , a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, busses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing devices 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. As an example, data 134 of memory130 may store predefined scenarios. A given scenario may identify a setof scenario requirements including a type of object, a range oflocations of the object relative to the vehicle, as well as otherfactors such as whether the autonomous vehicle is able to maneuveraround the object, whether the object is using a turn signal, thecondition of a traffic light relevant to the current location of theobject, whether the object is approaching a stop sign, etc. Therequirements may include discrete values, such as “right turn signal ison” or “in a right turn only lane”, or ranges of values such as “havingan heading that is oriented at an angle that is 30 to 60 degrees offsetfrom a current path of vehicle 100.” In some examples, the predeterminedscenarios may include similar information for multiple objects.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing devices 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. Similarly, thememory may be a hard drive or other storage media located in a housingdifferent from that of computing devices 110. Accordingly, references toa processor or computing device will be understood to include referencesto a collection of processors or computing devices or memories that mayor may not operate in parallel.

Computing devices 110 may all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes a seriesof microphones 152 or microphone arrays arranged at different locationson the vehicle. As shown, microphone arrays are depicted as separatefrom the perception system 172 and incorporated into the computingsystem 110. However all or some of microphones 152 may be incorporatedinto the perception system 172 or may be configured as a separatesystem. In this regard, the microphones may be considered independentcomputing devices operated via a microcontroller which sends signals tothe computing devices 110.

In one example, computing devices 110 may be an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may capable of communicating with various components ofthe vehicle. For example, returning to FIG. 1 , computing devices 110may be in communication with various systems of vehicle 100, such asdeceleration system 160 (for controlling braking of the vehicle),acceleration system 162 (for controlling acceleration of the vehicle),steering system 164 (for controlling the orientation of the wheels anddirection of the vehicle), signaling system 166 (for controlling turnsignals), navigation system 168 (for navigating the vehicle to alocation or around objects), positioning system 170 (for determining theposition of the vehicle), perception system 172 (for detecting objectsin the vehicle's environment), and power system 174 (for example, abattery and/or gas or diesel powered engine) in order to control themovement, speed, etc. of vehicle 100 in accordance with the instructions132 of memory 130 in an autonomous driving mode which does not requireor need continuous or periodic input from a passenger of the vehicle.Again, although these systems are shown as external to computing devices110, in actuality, these systems may also be incorporated into computingdevices 110, again as an autonomous driving computing system forcontrolling vehicle 100.

The computing devices 110 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 110 may navigate the vehicle to a destination locationcompletely autonomously using data from the map information andnavigation system 168. Computing devices 110 may use the positioningsystem 170 to determine the vehicle's location and perception system 172to detect and respond to objects when needed to reach the locationsafely. In order to do so, computing devices 110 may cause the vehicleto accelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 162), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 160), change direction (e.g., by turning thefront or rear wheels of vehicle 100 by steering system 164), and signalsuch changes (e.g., by lighting turn signals of signaling system 166).Thus, the acceleration system 162 and deceleration system 160 may be apart of a drivetrain that includes various components between an engineof the vehicle and the wheels of the vehicle. Again, by controllingthese systems, computing devices 110 may also control the drivetrain ofthe vehicle in order to maneuver the vehicle autonomously.

As an example, computing devices 110 may interact with decelerationsystem 160 and acceleration system 162 in order to control the speed ofthe vehicle. Similarly, steering system 164 may be used by computingdevices 110 in order to control the direction of vehicle 100. Forexample, if vehicle 100 configured for use on a road, such as a car ortruck, the steering system may include components to control the angleof wheels to turn the vehicle. Signaling system 166 may be used bycomputing devices 110 in order to signal the vehicle's intent to otherdrivers or vehicles, for example, by lighting turn signals or brakelights when needed.

Navigation system 168 may be used by computing devices 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 134 may store map information, e.g.,highly detailed maps that computing devices 110 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and right lane lines or other lane markersthat define the boundary of the lane. Thus, most lanes may be bounded bya left edge of one lane line and a right edge of another lane line.

The perception system 172 also includes one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 172 may include one or more LIDARsensors, sonar devices, radar units, cameras and/or any other detectiondevices that record data which may be processed by computing devices110. The sensors of the perception system may detect objects and theircharacteristics such as location, orientation, size, shape, type (forinstance, vehicle, pedestrian, bicyclist, etc.), heading, and speed ofmovement, etc. The raw data from the sensors and/or the aforementionedcharacteristics can be quantified or arranged into a descriptivefunction, vector, and or bounding box and sent for further processing tothe computing devices 110 periodically and continuously as it isgenerated by the perception system 172. As discussed in further detailbelow, computing devices 110 may use the positioning system 170 todetermine the vehicle's location and perception system 172 to detect andrespond to objects when needed to reach the location safely.

FIG. 2 is an example of map information 200 for a section of roadway.The map information 200 includes information identifying the shape,location, and other characteristics of various road features proximateto intersection 202. In this example, the map information 200information defining the shape and location of lanes 210-219 as well asshoulder areas 220 and 222. In this example, the map information 200includes information identifying lane lines 230-234 which define thelanes 210-212 as well as shoulder area 220. Lanes 211, 212, 215 and 216have the same direction of traffic flow (in an eastward direction),while lanes 210 and 217 have a different traffic flow (in a westwarddirection). Lanes 219 and 213 have a southward direction of trafficflow, and lanes 214 and 218 have a northward traffic flow. In addition,when a vehicle is traveling in lanes 211 and 212, solid line (white bar)240 may be used to define a bound of intersection 202 and/or a locationat which a vehicle which does not currently have a right of way to passthrough intersection from lane 211 or 212 must stop before proceeding.All this information may be included in the map information 200 for thearea proximate to intersection 202.

The map information may identify lanes or portions of lanes asindividual road segments which connect together in a grid or roadgraph.In this regard, given the simple nature of each of lanes 211-219, inthis example, each of these “lanes” as shown in FIG. 2 may be considereda road segment. Of course, the road segments of the map information mayactually be much smaller, for instance on the order of a few meters ormore or less.

Although the example of map information 200 includes only a few roadfeatures, for instance, lane lines, shoulder areas, an intersection, andlanes and orientations, map information 200 may also identify variousother road features such as traffic signal lights, crosswalks,sidewalks, stop signs, yield signs, speed limit signs, road signs, etc.Although not shown, the map information may also include informationidentifying speed limits and other legal traffic requirements, such aswhich vehicle has the right of way given the location of stop signs orstate of traffic signals, etc.

Although the detailed map information is depicted herein as animage-based map, the map information need not be entirely image based(for example, raster). For example, the detailed map information mayinclude one or more roadgraphs or graph networks of information such asroads, lanes, intersections, and the connections between these features.Each feature may be stored as graph data and may be associated withinformation such as a geographic location and whether or not it islinked to other related features, for example, a stop sign may be linkedto a road and an intersection, etc. In some examples, the associateddata may include grid-based indices of a roadgraph to allow forefficient lookup of certain roadgraph features.

FIGS. 3A-3D are examples of external views of vehicle 100. As can beseen, vehicle 100 includes many features of a typical vehicle such asheadlights 302, windshield 303, taillights/turn signal lights 304, rearwindshield 305, doors 306, side view mirrors 308, tires and wheels 310,and turn signal/parking lights 312. Headlights 302, taillights/turnsignal lights 304, and turn signal/parking lights 312 may be associatedthe signaling system 166. Light bar 307 may also be associated with thesignaling system 166. Housing 314 may house one or more sensors, such asLIDAR sensors, sonar devices, radar units, cameras, etc. of theperception system 172, though such sensors may also be incorporated intoother areas of the vehicle as well.

FIGS. 3A-3D also include microphones 152 (or 152 a-152 d) arranged atdifferent locations on the vehicle. These microphones may be considered“passive microphones” in that the microphones do not need to include anemitter (such as those used in sonar devices). In use, the actualspacing of the microphones should be reasonably close to expectedwavelengths of siren noises for the area (for instance, as differentstates and countries may use different siren noises) to avoid spatialaliasing or ambiguity. In this regard, given such frequencies, themicrophones may be located for example, on the order of ½ wavelength,apart from one another, in order to be able to compute direction fromthe relative phase of the sound waves that reach each microphone orrather the time difference of arrival. For instance, for emergencyvehicles in California, a 6 cm distance may be appropriate. Thisrelatively-small spacing may be achieved within a single set ofmicrophones or microphone array, such as microphones 152 a, arranged atthe front end of the vehicle 100. In that regard, as noted above,microphones 152 (including 152 a-152 d) may actually include sets ofmicrophones or microphone arrays. However, because microphones aredirectional, in other words an array on the front end of a vehicle doesnot hear sounds behind the vehicle well, more than one set ofmicrophones or array may be used. In this regard a second set ofmicrophones 152 b may be located at the rear of the vehicle 100.Additional microphone arrays, such as microphones 152 c and 152 d,oriented away from the sides of the vehicle (left and right) may also beused.

Although not shown in the FIGURES, in addition or alternatively,microphone arrays may be placed microphones around a roof panel of avehicle, such as around the circumference of the housing 314 (depictedhere as a dome). This may achieve both goals (arrays of closely spacedmicrophones oriented towards different directions relative to thevehicle) simultaneously, but the microphone arrays would have to beplaced in order to limit occlusion of sensors within the dome.

The instructions 132 may include a plurality of models for estimatingcharacteristics of siren noises. A first model may be configured todetect siren noise from any sounds received at the microphones. Forinstance, the output of the microphones may be input into the firstmodel in order to identify whether or not the output of the microphonesincludes a siren noise. In this regard, the first model may include amodel which provides a likelihood of the output of the microphoneincluding a siren noise for different types of noise.

The instructions 132 may also include a second model that can be used toestimate a bearing of a siren noise. For instance, the timing of thesiren noise reaching each of the microphones may be measured to providemeasurements as to a likely bearing, or relative direction, of thesource of the siren, or rather a probability distribution over possiblebearings.

The instructions 132 may also include a third model. This third modelmay use the microphone output, previously determined to include sirennoise by the first model, as well as timing and amplitudes of the sirennoise as input. With regard to the amplitudes, the presence andintensity of higher-frequency harmonics of a siren may also provide someindication of range, since the frequencies drop off at different rates.In some examples, the model may also use the estimated bearing andestimated range as input. In this regard, the third model may include amodel which uses all or some the aforementioned inputs to provide aprobability distribution over possible ranges (distances) of the sourceof the siren.

The instructions 132 may also include a fourth model. This fourth modelmay use the siren noise and timing collected over time to estimate aprobability distribution over possible relative velocities of the sourceof the siren noise. For instance, using the change in bearing over timemay provide an estimate of the relative velocity. In addition oralternatively, the model may include a neural net trained to predictlikelihood over relative velocities from a snippet of the siren sound.This snipped may be, such as 0.5 second, 1.0 second, 2.0 seconds, 3.0seconds, 4.0 seconds or more or less. The net may be able to extractrelative velocity from the change in amplitude as well as changes in theharmonics, and in some cases, from Doppler shifts of the sirenfrequencies.

One or more of the models described above may include learned models,for instance, those that utilize machine learning, such as classifiers.For instance, one or more classifiers may be used to detect the sirennoise, estimate a bearing, estimate a range, and estimate a relativevelocity. In other examples, rather than using all or some classifiers,the models may include one or more neural nets, such as those discussedabove to estimate relative velocities, or trackers, such as a Kalmanfilter or those that take in estimated bearings and estimated ranges,and/or corresponding probability distributions, over time, and outputother state estimates, such as estimated relative velocities. In stillother examples, estimated bearings may be determined using variousalgorithms such as a generalized cross correlation phase transform. Inanother example, estimated range may be computed analytically from theamplitude of the pressure sensed by the microphones because using theknowledge a range of siren volumes at a fixed distance and that pressurefalls off like 1/range.

Moreover, the examples described herein utilize four separate models,however, the models may be implemented as a single classifier to detecta siren noise, estimate a bearing, estimate a range, and estimate arelative velocity, or a plurality of models. For instance, a first modelmay detect a siren, and a second model may be used to estimate abearing, estimate a range, and estimate a relative velocity. In anotherinstance, a first model may detect a siren, a second model may be usedto estimate a bearing and estimate a range, and a third model may beused to estimate a relative velocity

Where needed to set up some of the models, some measure of ground truthdata may be extracted from a large set of logs. This may include, forinstance, manually labeled instances of real siren noises and no sirensas well as manually labeled or verified examples of which vehicle isgenerating the siren etc. At least some aspects of this labeling can beautomated using visual detection, such as by systems that utilizetemplates or image matching to identify particular types of objects fromcamera images or laser point clouds. For instance, if there is a labelthat a siren is present at time T, and at the same time the visualdetection identifies one and only one vehicle as being an emergencyvehicle with flashing lights, using an assumption that the vehicle wasthe source of the siren, the details of the vehicle, position, velocity,etc. over time, may be used to label the siren's relative and/orabsolute position, velocity, etc. over time.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

As noted above, computing devices 110 may control the movement ofvehicle 100 though its environment. FIG. 4 is an example of vehicle 100maneuvering on a portion of roadway 400 corresponding to the area of mapinformation 200. In this regard, intersection 402 corresponds tointersection 202, lanes 410-419 correspond to lanes 210-219, shoulderareas 420 and 422 correspond to shoulder areas 220 and 222, lane lines430-434 correspond to lane lines 230-234, and solid line 440 correspondsto solid line 240. Thus, in this example, vehicle 100 is approachingintersection 402 from lane 412 on a trajectory that will take vehicle100 into intersection 404 to proceed towards lane 415 or alternatively,to make a right turn into lane 413.

In this example, the vehicle's perception system 172 may provide thecomputing devices 110 with information about the vehicle's environment.This may include the location of objects such as lane lines 430-434 andsolid line 440, which have a corresponding feature in the mapinformation 200, as well as objects such as vehicles 451-454. Thecharacteristics of these different objects, including their shape,location, heading, velocity, etc. may be provided by the perceptionsystem 172 to the computing devices 110.

As the vehicle moves around, the output of the microphones 152 may befed into the first model order to detect potential emergency vehiclesirens. This may be done by the computing devices 110 or one or morecomputing devices of the perception system 172. Once a siren noise isdetected by the first model, further processing may be performed todetermine additional characteristics of the source of the siren noise.

For instance, the timing of the siren noise reaching each of themicrophones 152 may be measured and input into the second model toprovide measurements as to a likely bearing, or relative direction, ofthe source of the siren, or rather a probability distribution overpossible bearings. For instance, the microphones 152 may be timesynchronized in order to provide an estimated bearing of the source orwhat direction the siren is coming from relative to the vehicle. Thismay include a probability of the siren noise emanating from a pluralityof different directions around the vehicle (i.e. from 0 to 360 degreesaround the vehicle). The direction or range of directions, for instancea 5 degree or more or less range, with the highest probability may beconsidered to be an estimated bearing for the source of the siren noise.In addition or alternatively, the relative amplitude of the siren noisecan be used as an indication of bearing of a source of a siren noise.For example, a siren in front of the vehicle, may sound louder atmicrophones 152 a arranged at the front of the vehicle than atmicrophones 152 b arranged at the rear of the vehicle.

FIG. 5 provides an example representation 500 of bearing data for asiren noise determined using the timing as described above. Fordifferent bearings within a 360 degree radius around the vehicle, thebearing data may include a probability or likelihood value. In thisexample, the bearings are shown across ranges of 90 degrees representedby bearing ranges 510, 520, 530 and 540. Each of these bearing rangesrepresents a different relative direction around the vehicle: front(bearing range 510), left side (bearing range 540), rear (bearing range530), and right side (bearing range 520). Of course, larger or smallerranges may also be used, such as 180, 45 degrees, degrees, 30 degrees,15 degrees, 10 degrees, 1 degree, 0.5 degree, and so on. The likelihoodvalues may range on a scale of 0 to 1, 0 being less likely and 1 beingmore likely to represent a bearing of the source of the siren noise. Inthis example, bearing range 510 has a 0.10 likelihood value, bearingrange 520 has a 0.1 likelihood value, bearing range 530 has a 0.7likelihood value, and bearing range 540 has a 0.1 likelihood value.Thus, in this example, bearing range 530, having the highest likelihoodvalue may be selected or identified as an estimated bearing for thesource of the siren noise. In that regard, the source of the sirennoise, at least in the example representation 500, is likely locatedbehind vehicle 100.

In addition, the siren noise and timing may be input into the thirdmodel to provide a probability distribution over possible ranges (ordistances from the vehicle) of the source of the siren. Again, this maybe done by the computing devices 110 or one or more computing devices ofthe perception system 172. For instance, the timing for several secondsof output, such as 0.1 second, 1.0 second, 2.0 seconds, 3.0 seconds, 4.0seconds or more or less, from the microphones may be used to estimate arange of the source of the siren noise relative to the vehicle for aplurality of different ranges. Filtering this information over time mayprovide an improved estimated range. The range or a range of distances,such as 0.25 miles or more or less, across a plurality of differentranges, may be identified as an estimated range for the source of thesiren noise.

FIG. 6 provides an example representation 600 of range (distance) datafor a siren noise determined using the second model as described above.For different ranges, or really ranges of distances, from the vehicle,the range data may include a likelihood value. In this example, thedistances are shown across ranges of 0.25 mile represented by ranges0-0.25 mile, 0.25-0.50 mile, 0.50-0.75 mile, 0.75-1.00 mile, 1.00 mileor more. As noted above, the size of these ranges may be greater orsmaller (0.01 mile or 0.5 mile), defined in other scales (such asmeters), and so on. In addition, although the ranges in the example ofFIG. 6 extend to 1 mile or more, and the “top end” of a range ofdistances may be somewhat smaller, for instance 0.2 to 0.25 mile,depending upon the sensitivity of the microphone and accuracy achievableby the third model. In still other examples, the ranges may continuewell beyond 1 mile, for instance to 2, 5, 10, or more miles away fromthe vehicle.

Although depicted in FIG. 6 as being from the front of the vehicle, thedistances may be in any direction, including for instance the direction(or range of directions) for the estimated bearing. Again, thelikelihood values may range on a scale of 0 to 1, 0 being less likelyand 1 being more likely to represent a distance of the source of thesiren noise. In this example, range 0-0.25 mile has a 0.5 likelihoodvalue, range 0.25-0.50 mile has a 0.2 likelihood value, range 0.50-0.75mile has a 0.1 likelihood value, range 0.75-1.00 mile has a 0.1likelihood value, and range 1.00 mile or more has a 0.1 likelihoodvalue. Thus, in this example, range 0-0.25 mile, having the highestlikelihood value may be selected or identified as an estimated range forthe source of the siren noise. In that regard, the source of the sirennoise, at least in the example representation 600, is likely locatedwith 0.25 miles of vehicle 100.

In other examples, rather than providing likelihood values for ranges ofdistances, the third model may output an estimated range as a range ofdistances that meets a threshold likelihood or confidence value. Forexample, the third model may provide a range of distances that thatcorresponds to at least a 0.95 (or 95%) likelihood, or more or less, ofthe source of the siren noise being within that range of distances. Forinstance, for a source siren noise that is very nearby the vehicle, therange of distances may be on the order of 0 to 50 meters from thevehicle or more or less. For a source of a siren noise that is fairlydistant from the vehicle, the range may be, e.g, 100 to 400 meters fromthe vehicle or more or less. Because the pressure from sound waves of asiren noise hitting the microphones 152 drops off approximately by arate of the inverse of the distance from the microphones 152 to thesource of the siren noise (or 1/range), the range of distances whichmeet the threshold likelihood value (or confidence) will likely besmaller when the source of the siren noise is closer to the vehicle andlarger when the source of the siren noise is farther away from thevehicle.

These range estimates may be fairly inaccurate, with relatively largeerrors, for instance on the order of 1.5X to 2X (where X represents anestimated range or distance). However, despite these large errors, theestimated range may assist the vehicle's computing devices to determineif the source of a siren noise is too far away to react to (on the orderof a quarter of mile or more). At the same time, the second model mayalso provide an estimate of the range rate, or how fast the siren noiseis getting louder or softer. This may assist the computing devices 110in determining whether the source of the siren noise is getting closeror farther (towards or away estimate) which can be used to determine howto respond to the emergency vehicle. In the example of FIG. 6 , thetowards or away estimate may indicate that the emergency vehicle ismoving towards the vehicle 100.

The fourth model may also be used to estimate a probability distributionover possible relative velocities of the source of the siren noise usingthe siren noise and timing collected over time. In some examples,initial estimate may be compared to the map information in order torefine the estimate, for instance based on map constraints (such asspeed limits, etc.). Again, may be done by the computing devices 110 orone or more computing devices of the perception system 172. By filteringthis information over time may provide an estimated relative and/orabsolute velocity of the source of the siren noise.

FIG. 7 provides an example representation 700 of relative velocity datafor a siren noise determined using the third model as described above.For different velocities, or really ranges of velocities relative to thevehicle, the relative velocity data may include a likelihood value. Inthis example, the velocities are shown across ranges of 10 miles perhour relative to the vehicle represented by ranges less than −20 mph,−20-−10 mph, −10-0 mph, 0-10 mph, 10-20 mph, and greater than 20 mph. Ofcourse, additional ranges (20-30, −20-−30, 6 and so on) as well aslarger (20 mph) or smaller (1 mph) ranges may also be used. Moreover, asshown in the example of FIG. 7 , the ranges may even include negativevalues to indicate that the source of the siren noise is moving awayfrom the vehicle and positive values to indicate that the source of thesiren noise is moving towards the vehicle.

Again, the likelihood values may range on a scale of 0 to 1, 0 beingless likely and 1 being more likely to represent a velocity of thesource of the siren noise. In this example, range −20 or less has a 0.1likelihood value, range −20-−10 mph has a 0.1 likelihood value, range0-10 mph mile has a 0.5 likelihood value, range 10-20 mph has a 0.1likelihood value, and range 1 greater than 20 mph has a 0.1 likelihoodvalue. Thus, in this example, the relative velocity, having the highestlikelihood value may be selected or identified as an estimated relativevelocity for the source of the siren noise. In that regard, the sourceof the siren noise, at least in the example representation 700, islikely to be traveling at 0-10 mph relative to vehicle 100 and towardsvehicle 100 (as opposed to away or negative), or very close to the samespeed as vehicle 100.

In some examples, the first model may be used to identify exactly whatpart of the sound received at the microphone corresponds to a sirennoise. In other words, the first model may be used to identify whatsmall range of frequencies versus time correspond to a siren noise. Thiscan reduce the amount of information fed to the second, third and fourthmodels which is unrelated to the siren noise (i.e. interference fromsounds like wind noise or noise from nearby vehicles).

The information from the models as well as any of the estimatedcharacteristics may be provided to the one or more computing devices110. These computing devices may use the bearing data, range data,relative velocity data, estimated bearing, estimated range, towards oraway estimate, and estimated relative velocity to determine how thevehicle should react to the vehicle. Combining the examples of FIGS. 5-7, the computing devices may determine that the source of the siren noiseis located towards the rear of vehicle 100 (within bearing range 530),located at a range of 0-0.25 miles away from the vehicle 100, andtraveling at 0-10 mph relative to vehicle 100 “towards” (approaching)vehicle 100. However, to increase the usefulness of the response, theinformation provided by the models may be compared to objects detectedin the vehicle's environment to determine whether any of those objectsare the source of the siren noise.

For instance, as noted above, the perception system may detect andidentify objects within the range of the sensors of the perceptionsystem. Over time, the perception system may also determinecharacteristics of those objects, such as which of the objects arevehicles as well as the heading, location, and relative velocity of eachobject. This information may be compared with the estimated bearing,estimated range, and estimated relative velocity for the source of thesiren noise in order to identify which if any detected vehicles may bethe source of the siren noise. This may be done iteratively for everyidentified vehicle in order to produce a likelihood that it is thesource of the siren noise. At the same time, the computing devices mayproduce a likelihood that every identified vehicle is not producing thesiren. This may be an important value where the emergency vehicle is outof the range or otherwise occluded.

Returning to the example of FIG. 4 , comparing the estimatedcharacteristics of the source of the siren noise with the objectsidentified as vehicles as detected and determined by the perceptionsystem 172, the computing devices 110 may estimate a likelihood of eachof the detected vehicles being the source of the siren noise. Forinstance, the computing devices 110 may iterate through all of theidentified vehicles 450-454 and determine a likelihood value based onthe estimates that the source of the siren noise is located towards therear of vehicle 100 (within bearing range 530), located at a range of0-0.25 miles away from the vehicle 100, and traveling at 0-10 mphrelative to vehicle 100. The likelihood value of other vehicles in frontof the vehicle 100, or vehicles 452-454 is relatively low, for instance0.1 or more or less, since the source of the siren noise is locatedtowards the rear of vehicle 100. At the same time, the likelihood valuesof the vehicles towards the rear of vehicle 100, or vehicles 450 and451, is relatively high, for instance 0.5 or more or less, given theestimated bearing. The likelihood values of vehicles 450 and 451 mayalso differ based on the estimated range and estimated relativevelocity. For instance, the likelihood value of vehicle 450 may behigher than the likelihood value of vehicle 451, as vehicle 450 may bemoving at a velocity that is within 10 mph of the velocity of vehicle100 whereas the vehicle 451 may be traveling at a much higher relativevelocity, such as 20 mph greater than the velocity of vehicle 100.

While the example above relies on the data selected to be the estimatedbearing, estimated range, or estimated relative velocity, the likelihoodvalues for the objects in the vehicle's environment may alternatively bedetermined not by the data selected to be the estimated bearing,estimated range, or estimated relative velocity, but rather all of thebearing data, range data, and relative velocity data.

In some examples, the siren noise may be detected (for instance, usingthe first model as discussed above), before the source of the sirennoise is actually detected by the vehicle's detection system. In suchcases, once an object detected by the perception system is thenidentified as the source of the siren noise, prior model output (such asestimated bearing, estimated range, and estimated relative velocity),can be used to hypothesize where the emergency vehicle was coming from,for instance using the map information as well as information aboutareas within range of the perception system which may have previouslybeen occluded. This may be used to improve the estimate of which objector vehicle is the source (or is not the source) of the siren noise.

Once a particular vehicle is identified as the source of the sirennoise, the vehicle may be identified as an emergency vehicle. At thispoint, the observed movements of this emergency vehicle may also beconsidered when determining how best to respond to the emergencyvehicle, thereby further improving the usefulness of the response.Again, returning to the examples discussed above, given the likelihoodvalues, the computing devices 110 may identify vehicle 450 as the sourceof the siren noise. In this regard, the computing devices 110 mayobserve the movement of the emergency vehicle 450 and use thisinformation to determine how best to respond. Of course, in someexamples, there may be no other vehicles detected by the perceptionsystem 172 or all of the other vehicles may have likelihood values thatare too low or do not meet a minimum likelihood value threshold to beidentified as an emergency vehicles. In such cases, the computingdevices 110 may determine that the source of the emergency vehicle issimply not within range of the perception system 172 or is otherwiseoccluded, for instance, located behind an object such as anothervehicle, structure, etc.

In the example, of FIG. 4 , vehicle 100 may simply pull onto theshoulder area 420 and stop or slow down. Alternatively, if vehicle 100is traveling too quickly to safely pull onto the shoulder area 420, orif the vehicle was going to make a right turn at the intersection 404,the vehicle may continue to make the right turn into lane 413 if doingso would not impede the movement of the emergency vehicle 450 (such aswhere it would be faster and safe to turn than pull over because theshoulder area is very narrow, occupied, or ends quickly). In anotheralternative, if the shoulder area 420 is already occupied, it may besafer for vehicle 100 to change from lane 412 to lane 411 and allow theemergency vehicle 450 to proceed passed vehicle 100 and into theintersection 404.

In addition to comparing the estimated characteristics to informationfrom the perception system, these characteristics may be compared to mapinformation describing roadway features in the vehicle's environment.This may be used to identify a likely roadway, road segment or, in somecases, even a specific lane in which the emergency vehicle is traveling,again, even where the source is out of the range or otherwise occluded.For instance, the vehicle's computing devices may be able to identify aroad segment or segments ahead of the vehicle on the same road, a roadsegment or segments behind the vehicle on same road, road segment orroad segments on a crossing street, and event road segment or segmentson a road that does not intersect (or at least intersect nearby) thecurrent trajectory of the vehicle. Identifying a road segment mayinclude, for example, modeling probability over a plurality of roadwaysor road segments, not just identify the single most likely.

For instance, comparing the estimated bearing, estimated range, andestimated relative velocity to the example of FIG. 2 , the computingdevices 110 may determine that the source of the emergency vehicle ismost likely to be located on the road segments corresponding to lanes210, 211, and 212.

Using the map information may thus provide an even better estimate ofhow the vehicle should respond. Of course, if such map information isnot available, using cues about whether the emergency vehicle is infront of or behind, to the left or right, or approaching or recedingfrom the vehicle may also be useful.

In some instances, beamforming may be used to focus the microphones onlistening on locations that are most relevant to emergency vehicles.This may be done before or after a siren is identified. For instance,the map information and information from the perception system may alsobe used to beamform the microphones to focus listening on locations thatare most relevant to emergency vehicles. This may include, for example,roadways or near potential flashing light detections (as identified bythe perception system or by using information from the perceptionsystem). At the same time, beamforming may be used to ignoringinterfering sounds from other locations, such as tree rustling, windnoise, vibrations of the vehicle itself, nearby construction, etc. As anexample, for each array of the microphones, one beam may be formedstraight ahead, one offset 60 deg left, and one offset 60 deg right. Thefirst model may be used on the sound produced from each formed beam.Beamforming can increase signal-to-noise ratio quite substantially,which should, for example, increase the detection range of themicrophones. In addition, the beam in which the model gives the highestlikelihood of including a siren may be used as an indication of theapproximate bearing.

In addition, the output of the second model can be used to beamform themicrophones in order to focus on each direction in which there was apeak in probability of the siren noise emanating from that direction. Bydoing so, the increased signal-to-noise ratio may provide a moreaccurate estimation of the bearing of the siren noise. This, in turn,may provide for more accurate estimates of range and velocity.

As noted above, this additional processing may be useful in a variety ofcircumstances. The location of the vehicle relative to the emergencyvehicle (and vice versa) may be a significant factor in determining whattype of response is appropriate. As such if the emergency vehicle isbehind the vehicle, it may be most appropriate to pull over. In theemergency vehicle is oncoming or in front of and moving towards thevehicle, whether the vehicle should pull over or not depends on thephysical surroundings, such as whether or not there is a median.Similarly, if the source of the siren is coming from the side (left orright) of the vehicle as the vehicle approaches an intersection, thebest response may be to slow down dramatically or even stop before theintersection, even where the vehicle would otherwise be free to passthrough the intersection, such as when a light is green or when crosstraffic has a stop or yield sign. At the same time, if the sound iscoming from a neighboring, or for instance parallel street, respondingby changing behavior of the vehicle may not actually be appropriate.

Again, by having estimated characteristics of a source of a siren, thecomputing devices 110 may better control the reactive behavior of thevehicle 100. For instance, returning to FIG. 4 , if vehicle 450 isidentified as an emergency vehicle, the computing devices 110 maycontrol vehicle 100 to immediately pull over onto shoulder area 420,change to lane 411, or make a right turn as discussed in the examplesabove. If vehicle 451 is identified as an emergency vehicle, thecomputing devices 110 may control vehicle 100 to continue on the currenttrajectory at the same or slower speed or to pull over onto shoulderarea 420. If vehicle 452 is identified as an emergency vehicle, thecomputing devices 110 may control vehicle 100 to stop at solid line 440and/or before the intersection 404, even where the vehicle 100 otherwisewould have had the right of way to proceed through the intersection. Asan example instance, vehicle 100 may have the right of way where vehicle452 was subject to a stop sign and vehicle 100 was not or where vehicle100 currently has a green light (go) for intersection 404 and vehicle452 has a red line (stop) for intersection 404. If vehicle 453 isidentified as an emergency vehicle, the computing devices 110 maycontrol vehicle 100 to continue on the current trajectory at the same orslower speed. In this example, vehicle 100 should continue on thecurrent trajectory as vehicle 453 has already crossed over the vehicle100's trajectory. If vehicle 454 is identified as an emergency vehicle,the computing devices 110 may control vehicle 100 to continue on thecurrent trajectory at the same or slower speed. In this case, slowingdown and proceeding with caution may be an appropriate response whereinvehicle 100's trajectory follows vehicle 454, even though vehicle 454 ismoving away from vehicle 100, as the likelihood of sudden stops byvehicle 454 (due to a likely emergency) may be relatively high. Ofcourse, any number of additional maneuvers may be appropriate dependingupon the current circumstances of vehicle 100.

If none of vehicles 450-454 are likely to be the source of the sirennoise, or none of the other vehicles meet the minimum likelihood valuethreshold, again, the computing devices 110 may determine that thesource of the emergency vehicle is simply not within range of theperception system 172 or is otherwise occluded, for instance, locatedbehind an object such as another vehicle, structure, etc. Again, havingestimated characteristics, even without identifying a specific detectedvehicle as the source of the siren noise, may still provide thecomputing devices with useful information to determine how, if at all,to best respond to the siren noise. For instance, when an emergencyvehicle is occluded, such as when there are other vehicles between theemergency vehicle and the vehicle, the computing devices are still ableto recognize the siren to respond and pull over as needed.

In some cases, it can be difficult to resolve the bearing of the sirennose when there is another loud sound or interference (e.g. train,jack-hammer or other loud vehicle). When the interference is not beingboth at the same bearing and having high energy in the same frequenciesas the siren, various techniques may be used to focus the detection ofthe siren noise. One technique may include using beamforming asdiscussed above. If the siren noise and interference are at differentbearings, in a beam pointed at the siren, the siren noise will be muchlouder than the interference source compared to in data withoutbeamforming. In addition or alternatively, the bearing information maybe computed as a function of frequency and time. This bearinginformation, along with the amplitude information to the second model,so that the model can distinguish sounds that have similar frequencycontent, but different bearings. Again, in addition or alternatively,the bearing of a loud sound may be identified and classified using thefirst model. If the loud sound is not a siren (but rather,interference), a beam may be formed and used that passes sound exceptfor the bearing of the interference.

FIG. 8 is a flow diagram 800 that may be performed by one or moreprocessors such as one or more processors 120 of computing devices 110in order to detect and respond to emergency vehicles. In this example,at block 810, a plurality of microphones, such as microphones 152,arranged at different locations on a vehicle, such as vehicle 100, areused to detect a siren noise corresponding to an emergency vehicle. Theoutput from the plurality of microphones is used to estimate a bearingof the emergency vehicle and a range of the emergency vehicle at block820. The estimated bearing and estimate range are compared to mapinformation identifying the location of roadways subdivided into roadsegments in order to identify one or more road segments which theemergency vehicle could be traveling at block 830. At block 840, theestimated bearing and potential one or more road segments are used todetermine how to respond to the emergency vehicle. At block 850, thevehicle is controlled in an autonomous driving mode based on thedetermination of how to respond to the emergency vehicle.

As noted above, the operations described herein may be performed indifferent orders. For instance, estimated bearing and estimated range ofsounds received at the microphones 152 as a function of time andfrequency can be computed using the models (or other methods describedabove). These may then be fed into a model that actually detects oridentifies the siren noise.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

The invention claimed is:
 1. An autonomous vehicle comprising: aplurality of time-synchronized microphones; a classifier configured to:distinguish siren noise emanated by an emergency vehicle from soundsother than the siren noise detected by the plurality oftime-synchronized microphones based on amplitude and timing of thedetected sounds, and estimate characteristics of the emergency vehicle;and one or more processors operatively coupled to the plurality oftime-synchronized microphones and the classifier, the one or moreprocessors being configured to control the autonomous vehicle based onthe estimated characteristics of the emergency vehicle.
 2. Theautonomous vehicle of claim 1, wherein the estimated characteristicsinclude an estimated bearing of the emergency vehicle.
 3. The autonomousvehicle of claim 2, wherein spacing between the time-synchronizedmicrophones corresponds to a function of an expected wavelength of asiren noise that occurs in a geographic area in which the autonomousvehicle is used.
 4. The autonomous vehicle of claim 3, wherein thefunction is one half of a wavelength of the siren noise.
 5. Theautonomous vehicle of claim 2, wherein the estimated bearing is computedas a function of frequency, amplitude and time, so that different onesof the detected sounds that have similar frequency content, butdifferent bearings, can be distinguished from one another.
 6. Theautonomous vehicle of claim 1, wherein the estimated characteristicsinclude an estimated relative velocity of the emergency vehicle.
 7. Theautonomous vehicle of claim 1, wherein the estimated characteristicsinclude an estimated range of the emergency vehicle.
 8. A method,comprising; detecting, by one or more processors via a plurality oftime-synchronized microphones of an autonomous vehicle, a siren noise;detecting, by the one or more processors via a perception system of theautonomous vehicle, flashing lights of an emergency vehicle at a timewhen the siren noise is detected; estimating, by the one or moreprocessors, characteristics of the emergency vehicle based on thedetected siren noise and the detected flashing lights; and controlling,by the one or more processors, an autonomous vehicle based on theestimated characteristics of the emergency vehicle.
 9. The method ofclaim 8, wherein the autonomous vehicle is controlled to pull onto ashoulder area when the emergency vehicle moves toward the autonomousvehicle.
 10. The method of claim 8, wherein the autonomous vehicle iscontrolled to stop before an intersection when the emergency vehiclemoves toward the intersection.
 11. The method of claim 8, wherein theautonomous vehicle is controlled to make a turn based on a location ofthe autonomous vehicle relative to a location of the emergency vehicle.12. The method of claim 8, wherein the autonomous vehicle is controlledto continue on a current trajectory based on a location of theautonomous vehicle relative to a location of the emergency vehicle. 13.The method of claim 8, wherein when the emergency vehicle is locatedbehind an object, the flashing lights of the emergency vehicle areoccluded from being detected by the perception system of the autonomousvehicle.
 14. The method of claim 8, wherein the characteristics includean estimated relative velocity of the emergency vehicle.
 15. The methodof claim 8, wherein the characteristics include an estimated range tothe emergency vehicle.
 16. An autonomous vehicle comprising; a pluralityof time-synchronized microphones; a perception system; and one or moreprocessors operably coupled to the plurality of time-synchronizedmicrophones and the perception system, the one or more processors beingconfigured to: detect a siren noise via the plurality oftime-synchronized microphones; detect flashing lights of an emergencyvehicle at a time when the siren noise is detected via the perceptionsystem; estimate characteristics of the emergency vehicle; and controlthe autonomous vehicle based on the estimated characteristics of theemergency vehicle.
 17. The autonomous vehicle of claim 16, wherein theautonomous vehicle is controlled to pull onto a shoulder area when theemergency vehicle moves toward the autonomous vehicle.
 18. Theautonomous vehicle of claim 16, wherein the autonomous vehicle iscontrolled to stop before an intersection when the emergency vehiclemoves toward the intersection.
 19. The autonomous vehicle of claim 16wherein the autonomous vehicle is controlled to make a turn based on alocation of the autonomous vehicle relative to a location of theemergency vehicle.
 20. The autonomous vehicle of claim 16, wherein theautonomous vehicle is controlled to continue on a current trajectorybased on a location of the autonomous vehicle relative to a location ofthe emergency vehicle.